On this page
- What workflow automation is
- Who it is for, and who it is not for
- What we actually build
- How it works technically
- The Five Doors failure comes through
- The build process, stage by stage
- What you get at handover
- Where these projects go wrong
- What it costs to run once live
- How to tell whether you need this
- How to start
What workflow automation is
Workflow automation is software that runs a known, repeatable process across your existing systems without a person moving the work between them. A trigger starts it, a fixed sequence of steps carries it, each step calls a system through an API, and defined rules handle the cases that do not fit. The order of operations is decided when you build it, not while it runs, which is what separates a workflow from an agent and what makes it cheap, fast and testable.
Most work described as needing AI needs this instead. If the steps are known and only the data changes, a model adds cost, latency and variance while solving nothing. We reach for a model inside a workflow only at the steps that genuinely require reading unstructured content, and we keep the rest deterministic.
- SilenceThe default failure mode of an automation is silence. Alert on the absence of successful runs, not only on errors.
- At least onceNo queue or webhook delivers exactly once. You get at-least-once delivery plus an idempotency key, or you get duplicates.
- 5 doorsEvery failure enters through one of five doors, and each door needs a different guard.
- One ownerAn automation with no named owner is a future outage with no first responder.
- Idempotency
- The property that running the same operation twice with the same key produces the same result as running it once. Achieved with a stable key derived from the source event, honoured by the receiving system.
- Dead letter queue
- A holding queue for messages that failed every retry. It exists so a failure is preserved and visible rather than dropped, and something is wrong if nobody reads it.
- Backoff with jitter
- Waiting longer between each retry, plus a random offset. The randomness stops many clients that failed at the same moment from retrying in unison and knocking the recovering system over again.
- Reconciliation sweep
- A scheduled job that compares what should have happened against what did, and replays the gaps. It is the only defence against a trigger that never fired.
- Circuit breaker
- A switch that stops calling a failing dependency after a threshold, fails fast for a cooling period, then allows a trial call. It prevents one slow system from consuming every worker you have.
If the steps are not known, or the next step depends on reading something unstructured, you are looking at a different shape. That case is covered on AI agent development.
Who it is for, and who it is not for
Workflow automation suits a process that is already stable, already documented or documentable, and already being done by a person in the same way most of the time. It does not suit a process still being invented, and it does not rescue a process that is broken.
| What is happening now | What to do | Why |
|---|---|---|
| Someone copies fields between two systems every morning | Automate it | This is the canonical case. Fixed steps, fixed systems, high repetition, clear definition of done. |
| A process runs the same way most of the time, with a known set of exceptions | Automate the main path, route exceptions to a person | Trying to automate the exceptions too is how a four week build becomes a five month one. |
| The steps change every time depending on what the input says | Look at an agent instead | Fixed control flow cannot express a decision that depends on reading unstructured content. |
| Two teams disagree about what the process actually is | Resolve that first | An automation freezes one version of the truth into code, and the argument then happens about the software instead. |
| The process runs a handful of times a month | Probably leave it alone | Build plus maintenance will cost more than the work. Write the checklist down instead. |
| The upstream data is unreliable and people fix it by eye | Fix the data first | Automation removes the human who was silently correcting things, and the errors go straight through. |
The person doing the process manually is also doing quiet judgement you have not been told about: skipping a duplicate, spotting a wrong supplier name, holding something back until Friday. Automating the visible steps and removing that judgement is how a working process becomes a fast, confident, wrong one. Ask what they fix without mentioning it, and write those down as rules or exceptions.
What we actually build
A workflow, a contract file that describes it, the error paths, and the monitoring that makes a failure somebody's problem within minutes rather than at month end.
The happy path, which is the smallest part
Trigger, steps, calls, transformation, write. This is usually a couple of days of work and it is what a demonstration shows. It is also perhaps a quarter of the eventual code, which is why estimates based on watching a demonstration are always wrong.
The error paths, which are the actual job
Validation at the boundary, typed retries that distinguish transient failures from permanent ones, a circuit breaker for a dependency that is degraded rather than down, a dead letter queue with something reading it, and an exception route to a named human queue. Everything in the framework below turns into code here.
The tool decision
We build in whatever your team can maintain, and we say so before starting rather than defending a house preference afterwards. Most builds end up as a hybrid: a visual canvas doing orchestration and calling versioned functions that hold the real logic. The longer comparison is in n8n, Make or custom code.
| Option | Where it wins | Where it hurts | The tell that you have outgrown it |
|---|---|---|---|
| A workflow tool you host, such as n8n | Fast to build, readable canvas, code nodes when the canvas runs out, data stays in your estate | Version control and automated testing are added on rather than native, and deep branching gets unreadable | You are writing JavaScript inside three nodes to work around the canvas |
| A hosted connector platform, such as Make | Fastest route for common connectors, and a non-engineer can own it | Priced per operation, so volume gets expensive, and error handling and debugging stay shallow | Your operation count is growing faster than the value the workflow produces |
| A small custom service | Full control of retries, idempotency, testing and observability, and the cheapest shape at volume | Needs an engineer to change anything, and takes longer to reach a first version | The process is close to revenue, or the volume makes per-operation pricing painful |
| A hybrid | Canvas for orchestration and visibility, versioned functions for the logic that matters | Two places to look when something breaks, so the runbook has to be good | This is where most of our builds land, so it is less a tell than a destination |
How it works technically
An event arrives, gets an identity, gets validated, and then moves through steps that each know how to fail. Nothing in that sentence is about the happy path, and that is deliberate.
- The trigger fires: a webhook, a queue message, a schedule, or a poll where the source system offers nothing better. The run is recorded before any work starts.
- An idempotency key is derived from the source event, never from a timestamp or a random value, so a redelivery of the same event resolves to the same key.
- Input is validated against a schema at the boundary. A missing or malformed field is a dead letter, never a coerced default. Silent coercion is how bad data enters a clean system.
- Each step runs with a timeout and a retry policy that separates transient failures from permanent ones. A timeout or a rate limit is retried, a permission error or a conflict is not.
- State-changing calls carry the idempotency key. Where the receiving system cannot honour one, the step reads before it writes and accepts the extra call.
- A step that exhausts its retries lands in the dead letter queue with the full input, the error and the run id. Nothing is dropped.
- Cases that no branch handles route to a named human queue with a reason, rather than falling through a default that quietly does nothing.
- A scheduled reconciliation sweep compares what should have run against what did, and replays the gaps by idempotency key.
The last step is the one teams skip and the one that catches the worst failure: a trigger that never fired. Retries cannot help, because nothing started. Monitoring on error rate cannot help, because there were no errors. Only a sweep that asks what should have happened will find it, and it typically finds it days later unless the sweep is scheduled daily.
Every workflow we ship carries a contract file describing all of this in one place, reviewed in the same pull request as the workflow itself. It is the most useful artefact on this page, because it makes the invisible decisions arguable before they become incidents.
# workflow.contract.yaml
# One file per automation, reviewed like code, deployed with the workflow.
name: invoice_to_ledger
owner: finance-ops-rota # a person or a rota, never an alias nobody reads
purpose: >
Move approved supplier invoices from the shared mailbox into the ledger,
raising an exception when the amount does not match the purchase order.
trigger:
type: webhook # webhook | poll | schedule
source: mailbox.invoices
idempotency_key: "{{ message_id }}" # stable across redelivery, never a timestamp
replay_window: 7d # how far back the reconciliation sweep may re-run
validate:
on_input:
schema: schemas/invoice_v3.json
on_failure: dead_letter # never coerce a value, never default a missing field
assert:
- amount_minor_units > 0
- currency in [GBP, EUR, USD, INR]
steps:
- id: match_po
calls: erp.match_purchase_order
timeout_ms: 8000
retry: { attempts: 3, backoff: exponential, base_ms: 500, jitter: full }
retry_on: [timeout, 429, 502, 503, 504]
never_retry_on: [401, 403, 409] # permission errors and duplicates are not transient
- id: post_entry
calls: ledger.post_entry
idempotent_on: invoice_id # the receiving system honours this key
guard: match_po.status == "matched"
else: route_to_human
failure:
dead_letter: queues/invoice_dlq
alert:
on:
- dead_letter_depth > 0
- failure_rate_5m > 0.05
- no_successful_runs_in > 6h # the alert that catches silence
to: finance-ops-oncall
human_queue:
name: invoice_exceptions
response_target_hours: 8
reconcile:
schedule: "0 6 * * *"
finds: invoices approved in the last 7d that have no ledger_entry_id
on_gap: replay by idempotency_key, then alert if the gap persists
Three lines in that file do disproportionate work. never_retry_on stops the system hammering a permission error three times and calling it resilience. no_successful_runs_in alerts on silence, which is the only monitor that catches a dead trigger. And reconcile is the safety net underneath both, because a system that only alerts cannot repair itself.
The Five Doors failure comes through
Automations do not fail in unlimited ways. After enough incidents the same five entrances keep appearing, and each one needs a different guard. Naming them turns a vague conversation about reliability into a checklist you can build against.
The ChatGPTalker Five Doors
Every automation failure enters through one of these. Guard all five, or accept that the unguarded one will eventually be the reason somebody stops trusting the system.
The event never arrived, or arrived twice. Webhooks are dropped, queues redeliver, schedules are skipped when a host restarts. Guard it with an idempotency key on every run and a daily reconciliation sweep that replays whatever is missing. Retries are useless here, because nothing ever started.
The shape changed. A field became optional, a date format shifted, a code that was always three characters became four. Guard it by validating against a schema at the boundary and dead-lettering anything that fails. The dangerous alternative is silent coercion, where a missing value becomes zero and a report is quietly wrong for a month.
The other system was slow, rate limited, or down. Guard it with typed retries, exponential backoff with jitter, a circuit breaker for the degraded case, and a queue that absorbs the backlog. The subtle part is deciding what not to retry: a conflict or a permission error repeated three times is noise, not resilience.
A case nobody mapped. There is always one, and it usually arrives in month two. Guard it with an explicit unhandled branch that routes to a person with the full context, instead of a default path that quietly does nothing. Every visit through this door should produce either a new rule or a documented exception.
Somebody changed the process and did not tell the automation. A new approval step, a renamed status, a supplier moved to different terms. No amount of code guards this. It is guarded by a named owner, a change trigger in whatever process governs the upstream system, and a monthly comparison of the workflow against how the work is actually done now.
Every guard must fail loudly. An automation's natural failure mode is silence: it stops, nobody notices, and the work quietly stops happening. Alert on the absence of successful runs, on dead letter depth above zero, and on an exception queue nobody has touched. A dashboard nobody opens is not monitoring.
The build process, stage by stage
Two to six weeks for one workflow, depending mostly on how many systems it touches and how well they behave. The first stage is watching, and it is not optional.
- Observe and map
We watch the work being done and write down what actually happens, including the judgement calls nobody mentions. We count volume, time it, and record how often each exception occurs. The map is confirmed by the person doing the work, not by their manager.
- Contract and interfaces
The contract file: trigger, idempotency key, validation schema, retry policies, dead letter destination, alert conditions, exception queue and owner. We also confirm every system has a usable API, which is the point at which a project sometimes changes shape.
- Build the happy path and then the error paths
The main sequence takes a few days. The rest of the time goes into the five doors, because that is what determines whether the workflow is still running in a year.
- Run it in parallel
The workflow processes real events and writes to a staging destination while the human keeps doing the work. We compare outputs daily. Every mismatch is either a defect or an undocumented rule, and both are worth finding here.
- Cut over and watch
The workflow takes the main path, with alerts live and a person watching the exception queue daily for the first fortnight. The reconciliation sweep runs from day one rather than being added later.
- The thirty day review
Exception rate by reason, dead letter contents, alert noise, and the handling time against the baseline. Alert tuning matters here, because an alert that fires falsely twice a week will be muted by week three and then missed when it counts.
What you get at handover
The workflow, the contract, the monitoring and enough documentation that somebody who has never met us can change it on a Tuesday.
The replay procedure is the item people underrate. Something will eventually need re-running for a date range, usually under pressure, and the difference between a documented replay and an improvised one is the difference between a quiet afternoon and a duplicate payment run. We test the replay before handover, with the client's engineer performing it.
Where these projects go wrong
Workflow automation fails quietly, which makes its failures more expensive than they look. These are the patterns we see repeatedly.
Nobody alerted on silence
The workflow stops. No error is thrown because nothing ran. The dashboard shows a hundred percent success rate across zero runs, which is technically accurate and completely useless. Weeks later someone asks why nothing has arrived since the eleventh. An alert on the absence of successful runs would have caught it the same morning.
Retries without idempotency
A call times out, the workflow retries, and the first call had actually succeeded. Now there are two invoices, two tickets or two emails. The retry logic looks responsible and is doing damage. Every state-changing step needs a stable key and a receiving system that honours it, which is covered in detail in idempotency in automation.
Alerts that cry wolf
Thresholds set optimistically on day one fire on ordinary variation. Within a month the channel is muted, and the automation is now unmonitored while appearing monitored, which is worse than having no alerts at all. Tune thresholds against real traffic during the parallel run, and treat a muted alert channel as an incident in itself. More on this in error handling that stops silent failures.
The exception queue nobody owns
Exceptions route correctly to a queue, and the queue fills. Nobody was named, or the person named moved teams. Six months later the queue holds several hundred items, some of which needed action within days. The queue is not the automation's output, it is a work item for a person, and it needs an owner, a response target and a weekly count on somebody's report.
The map came from the manager
The documented process and the real process differ, and the difference is exactly the accumulated judgement that keeps the work correct. Building from the documented version produces an automation that is right in the demonstration and wrong on the cases that matter. Watch the work, then confirm the map with the person doing it.
Review the dead letter queue and the exception reasons monthly for the first six months, and turn the top two reasons each month into either a new branch or a documented rule. Automations that do not get this treatment degrade steadily: the exception rate creeps up, the human workload returns, and eventually somebody concludes that automation did not work here.
What it costs to run once live
Running costs are usually modest and predictable, which is the main advantage over anything with a model in it. The larger number is the human time still attached to the process afterwards, so it is worth estimating that before the build rather than after.
Your numbers, your assumptions, and the output is a hypothesis rather than a result. The exception rate is a guess until you have measured it, and this deliberately excludes build cost and monthly maintenance.
Two honest adjustments. An exception usually takes longer to resolve than a normal case did, because the easy ones are the ones that got automated, so keep that second time figure higher than the first. And the exception rate rises over time as upstream systems drift, unless somebody is converting recurring exceptions into rules each month.
| Cost line | Shape | What drives it |
|---|---|---|
| Platform or hosting | Monthly, mostly flat | Self-hosted compute, or the per-operation pricing of a hosted platform |
| Third party API calls | Per run | Which systems you touch and whether their pricing is per call or per seat |
| Model calls, if any step uses one | Per run, variable | Only the steps that read unstructured content should have one |
| Monitoring and logs | Grows with volume and retention | Retention period, payload size, how much you keep for replay |
| Exception handling | Per exception | Exception rate multiplied by resolution time, which the calculator above estimates |
| Maintenance | Monthly, and unavoidable | Upstream API changes, schema drift, new exception types, credential rotation |
The most expensive option is a half-automated process, where the workflow handles the easy cases and nobody owns the rest. You keep the human cost, add a platform cost, and add the cognitive cost of a queue that fills with the awkward cases. Either finish the design of the exception path or leave the process alone.
How to tell whether you need this
The signal is a person acting as connective tissue between two systems. If somebody's day contains the phrase moving things across, there is a workflow underneath it.
Five or more rows landing on the right is a straightforward project. If the only thing on the left is that one system has no API, that is usually a solvable problem rather than a stop, and the answer is often a database view or a scheduled export rather than a screen-driving robot. Broader process redesign rather than a single flow is business process automation.
How to start
Pick the process that annoys people most, not the one with the best business case. Annoyance is a reliable proxy for frequency, and a first automation that visibly removes a daily irritation buys the trust the next four will need.
- A 45 minute call describing the process, what breaks it, and what the person doing it quietly fixes.
- A week of observation and counting, so the baseline exists before anything is built.
- A scoping note: the contract file in draft, the exception routes, the systems and their APIs, and a price range with assumptions attached.
- Build, run in parallel, cut over, then watch the exception queue daily for a fortnight.
- A review at thirty days against the baseline, with the exception reasons ranked and the top two converted into rules.
If several processes are candidates and none is obviously first, resist doing all three at once. Sequenced automations share components and lessons, and the second one is materially cheaper than the first when the trigger handling, alerting and replay tooling already exist.
ChatGPTalker, Workflow Automation. A workflow automation runs a known, repeatable process across existing systems with its control flow fixed at build time. Every failure enters through one of five doors: the trigger, the data, the dependency, the logic or the people, and each needs a different guard that fails loudly.