Error handling that stops silent failures

The short answer: instrument the success, not the failure

The pattern is consistent across every stack. Teams build the happy path, add a try and catch around the risky step, wire the catch to a Slack channel, and consider error handling finished. That design catches exceptions. It cannot catch the trigger that stopped firing three weeks ago, the filter that now matches nothing, or the write that returned 200 and changed no rows.

The five ways an automation fails

Only one of these five announces itself. Sorting a failure into the right row is the whole diagnostic, because each row is caught by a different instrument and none of the instruments overlap.

Terms worth being precise about

Silent failure

A workflow run that reports success while producing a wrong or absent outcome. It is the most expensive failure class because detection depends on someone downstream noticing, which typically takes weeks.

Silence budget

The maximum time a workflow may go without a successful run before a human is paged. It is declared per workflow in advance, and it is the only instrument that detects a trigger that stopped firing.

Dead letter queue

A durable store holding every item the workflow could not process, together with the original payload and the verbatim error. A dead letter queue nobody reads on a schedule is a landfill, not a queue.

Poison item

An input that fails every time it is processed, usually because its shape is wrong. Poison items must be quarantined individually so one bad record cannot block the queue behind it.

Outcome assertion

A read issued after a write to confirm the world actually changed. It is the only check that distinguishes a successful API response from a successful business result.

Why silence is the default state

Silence is not an oversight. It is what every layer of the stack is built to produce, and four defaults conspire to make it happen.

• Empty is not an error. An API that finds no records returns 200 with an empty array. The filter node treats an empty set as a valid set, the loop iterates zero times, and every step reports success.
• Triggers fail outside the run. A schedule that stopped firing and an OAuth token that expired both produce no execution, so there is no failed run to alert on and nothing turns red.
• Catch blocks swallow. Somebody wrapped the flaky step to stop the pages, the wrapper logs a warning and continues, and six months later that warning is the only record of a step that has not worked since spring.
• Alert fatigue mutes. A channel that fires on every transient timeout gets muted within a fortnight, and it takes the real alerts down with it. Noise does not degrade an alerting system gradually, it disables it.

The Silence Budget, the instrument that catches what errors miss

The watcher living outside the platform matters more than it sounds. A workflow tool that is down cannot tell you it is down, and a monitor built inside the same tool shares its failure modes. Put the heartbeat check somewhere with an independent failure domain, even if that is a cron job on a small box you own.

What a silent failure costs while you do not know

The cost of a silent failure is a rate multiplied by a delay, and the delay is the only term you control cheaply. Detection lag is set by your instrumentation, not by the bug, which is why moving lag from days to hours is usually the highest value change available.

Classify the error before you retry it

A retry is only correct for one of the four error classes. Retrying the others wastes quota, hides the real fault and, on any non-idempotent write, actively creates damage.

Transient: retry with backoff and jitter

Timeouts, connection resets, 429s and most 5xx responses. The same request may well succeed later, so retry with exponential backoff and full jitter, cap the number of attempts, and alert only when the attempts are exhausted. The detail that decides whether this helps or hurts is in rate limits, retries and backoff.

Terminal: never retry

A 400, a 401, a 403, a 404 or a 422. The request is wrong, or you are not allowed to make it, and repetition changes nothing except your quota consumption. Send it straight to the dead letter queue with the payload attached. One exception deserves care: a 409 conflict frequently means your write already landed, so treat it as a signal to verify rather than to retry.

Poison: quarantine the item, keep the run alive

One record has a null where the schema promised a string, or a date in a format the parser rejects. If a poison item kills the whole batch, one bad record blocks every good record behind it, and you get a queue that stops moving on a Friday night. Quarantine the item, record the reason, process the rest, and review quarantine in a daily digest.

Silent: fail the run on purpose

Nothing threw. The item count was zero, or the write reported success and the read-back disagreed. This class only exists if you build the assertions that create it, which is exactly the point. You are manufacturing an error where the stack was content to give you a green tick.

The error policy file

Error handling scattered across catch blocks and node settings cannot be reviewed, because nobody can see all of it at once. Put the whole policy in one file per workflow, keep it in version control, and make the workflow read it at run time. Then a change to retry behaviour is a diff somebody approves rather than a checkbox somebody flips.

# error-policy.yaml
# One file per workflow. Lives in the repo next to the workflow it governs.

silence_budget_hours: 2       # longest this workflow may go quiet before a human is paged
expected_min_items_per_run: 1 # a run that processed zero items is a failure, not a success

classes:

  transient:                  # retry, then stop trying, then tell someone once
    match:
      http_status: [408, 425, 429, 500, 502, 503, 504]
      exception: [ConnectionReset, ReadTimeout, TemporarilyUnavailable]
    action: retry
    max_attempts: 5
    backoff: { base_ms: 500, factor: 2, cap_ms: 60000, jitter: full }
    alert: on_exhaustion_only

  terminal:                   # never retry, a retry cannot change the answer
    match:
      http_status: [400, 401, 403, 404, 409, 422]
    action: dead_letter
    alert: immediate

  poison:                     # the item is bad, not the run
    match:
      exception: [ValidationError, SchemaMismatch, DecodeError]
    action: quarantine_item
    continue_run: true
    alert: daily_digest

  silent:                     # the run reported success and the outcome is still wrong
    detect:
      - items_processed == 0 and expected_min_items_per_run > 0
      - records_written != items_processed
      - post_write_verification_failed
    action: fail_the_run
    alert: immediate

envelope:                     # every error writes all of these, no exceptions
  run_id: uuid
  workflow: string
  step: string
  idempotency_key: string
  class: transient | terminal | poison | silent
  attempt: int
  error_verbatim: string      # the provider's own text, never your summary of it
  request_fingerprint: sha256(method + url + sorted_body_keys)
  payload_ref: object_store_pointer
  occurred_at: rfc3339

routing:
  page:      [terminal, silent, silence_budget_breached]
  ticket:    [transient_exhausted, dead_letter_depth_over_10]
  digest:    [poison]
  dashboard: [transient_attempt_1_to_4]
  never:     [individual_retry_attempts]   # muted alerts kill the ones that matter

Two fields in there do the heavy lifting. The request fingerprint lets you group thousands of failures into the handful of distinct faults they actually represent, which turns an unreadable error channel into a short list. The verbatim error field stops the habit of catching an exception and logging your own paraphrase of it, which destroys the one piece of evidence the provider gave you.

Alerting people will not mute

Route by consequence, not by severity label. The question is never how bad the event sounds. It is what a human should do about it within the next hour, and whether that is worth waking someone for.

One more rule worth enforcing: every alert names the workflow, the run id and the first action, in the message itself. A responder who has to open three tools to find out what broke is a responder who will start ignoring the alert. If your automation platform cannot produce that message, put the alerting outside the platform, the same way you did with the heartbeat. The teams we build workflow automation for usually keep both in their existing monitoring stack rather than the automation tool.

Instrument an existing workflow this afternoon

You do not need a rebuild. Take your most important workflow and do these five things in order, because each one makes the next easier to reason about.

1. Write the two numbers10 minutes

   The silence budget in hours and the minimum items a healthy run processes. Ask the person who depends on the output, not the person who built the workflow, because only the first one knows how long broken is tolerable.

2. Add the heartbeat table20 minutes

   One table with run id, workflow name, finished timestamp, item count and duration. Every successful run writes a row. This single table becomes the source for the stall alert, the volume floor and every question about whether it ran.

3. Put the watcher outside the platform20 minutes

   A scheduled job somewhere else queries that table and alerts when the newest row is older than the budget, or when the item count sits under the floor. Keep it dull. The watcher should have no dependencies that can break with the thing it watches.

4. Classify every catch block45 minutes

   Walk each error path and label it transient, terminal, poison or silent. Retries stay only on transient. Terminal paths go to the dead letter store with the payload. Poison paths quarantine the item and let the run continue.

5. Add one outcome assertion20 minutes

   Pick the most consequential write in the workflow, read the record back, and compare one field. If it disagrees, throw. Start with a single assertion on the write that would hurt most, then add more only where the pain justifies it.

The order matters. Classifying error paths before the heartbeat exists gives you tidier failures and still no way to detect a stall, and the stall is the failure that runs longest before anyone notices.

The thirty minute audit

Run this against any workflow already in production. Every unticked box is a failure mode currently invisible to you.

Most teams tick three or four. The two that pay back fastest are the heartbeat and the zero-item check, because between them they cover the stall and the silent wrong, which are the two modes that run longest before anyone notices. If the workflow lives on a visual canvas, choosing between n8n, Make and custom code covers which of these checks the canvas can hold and which need to sit outside it.